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Background / Context: 

Empirical evidence confirms that increased education is positively associated with higher 
earnings across a wide spectrum of fields and student demographics (Barrow & Rouse, 2005; 
Card, 2001; Cameiro, Heckman, & Vytlacil, 2011; Dadgar & Weiss, 2012; Dynarski, 2008; 
Jacobson & Mokher, 2009; Jepsen, Troske, & Coomes, 2009; Kane & Rouse, 1995; Marcotte, 
2004). College degree holders earn higher salaries and experience unemployment less frequently 
than those who do not have college degrees (Dadgar & Weiss, 2012), and those who attended 
college, regardless of degree receipt, report higher rates of job satisfaction, promotion 
opportunities, increased work responsibilities, and improved work performance (Hoachlander, 
Sikora, & Horn, 2003). 

For many low-income students, the pathway to higher education and increased earnings 
begins in community colleges, which currently enroll about one-third of American degree- 
seekers (Digest of Education Statistics, 2012). However, community college students’ 
persistence and completion rates leave much room for improvement. Among students beginning 
at public two-year colleges, fewer than half earn a credential or transfer to a four- year institution 
within six years after their initial enrollment (Radford, Berkner, Wheeless, & Shepherd, 2010). 
Many community college students stop or drop out long before graduation day, losing out on the 
full complement of higher earnings and employment outcomes associated with higher education. 

One factor contributing to these low success rates is the fact that many students arrive on 
campus only to find that they are required to take non-credit-bearing courses, called 
“developmental” or “remedial” education courses, prior to enrolling in college-level courses 
(Duke & Strawn, 2008). One estimate of the prevalence of students required to complete 
developmental coursework suggests that among students whose first institution attended was a 
community college, around 60 percent took a remedial course at a postsecondary institution 
(Adehnan, 2004; Bailey, Jeong, & Cho, 2009). Unsurprisingly, degree or certificate attainment 
rates among students who need developmental education are even lower than those of the general 
population (Attewell, Lavin, Domina, & Levey, 2006). Students in need of developmental 
education are not just failing to earn a credential - the majority never complete the 
developmental course sequence to which they are referred (Bailey 2009). One popular strategy 
aimed at addressing the issues of students referred to developmental courses (as well as students 
in college-level courses) is called “learning communities,” or LCs. 

The LC evaluated in this study produced positive short-term impacts on academic 
outcomes including credits attempted and credits earned and students’ progression through the 
developmental reading sequence of courses. Given these positive results, similar learning 
communities for developmental reading and math students in community colleges were 
replicated and evaluated at five other schools. 

Purpose / Objective / Research Question / Focus of Study: 

Although LCs have been popular for decades, there have been few experimental tests of 
their efficacy. A variety of studies have looked at LC students’ outcomes compared to a 
constructed comparison group, but generally, comparisons were made between students who 
were in learning communities and those who were not, despite the fact that students typically 
self-select into learning communities and are not directly equivalent. The first large sample 
random assignment evaluation of learning communities was conducted by MDRC in 2003-2005 
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to evaluate the effectiveness of Kingsborough Community College’s (KCC) learning 
communities program. The primary objective of this study is to estimate the causal effect of the 
opportunity to participate in a learning communities program at KCC. Program effects are 
estimated for academic (credit accumulation and degree completion) and economic (employment 
and earnings) indicators. 

Setting: 

Kingsborough Community College, located in Brooklyn, New York, is part of The City 
University of New York (CUNY) system, the largest urban public university system in the 
country. KCC is has about 18,000 students, of whom about 75% receive financial aid. KCC 
offers a one-semester learning community in which all courses are linked together. 

Population / Participants / Subjects: 

The study randomly assigned over 1,500 students to either a program group, which had 
the opportunity to participate in the Opening Doors Learning Community, or a control group, 
which could enroll in the colleges’ usual unlinked courses. More than two-thirds of the sample 
required developmental English at the start of the study (and the learning communities to which 
they were assigned included a developmental English course as one of the linked courses). 

Intervention / Program / Practice: 

Learning communities involve the co-enrollment of a cohort of students into two or more 
courses. Typically, the curricula of these courses are intentionally linked or integrated, 
sometimes around a theme. For example, a learning community with an overarching theme and 
title called “Poverty and Inequality” might link a developmental English course and a college- 
level sociology course. The courses could require that students learn various elements of essay 
writing and argumentative rhetoric in their English course and then use those techniques to write 
a paper in their sociology course, exploring the relationship between income status and race. 
Learning communities tend to require faculty collaboration, which enables teachers to 
communicate about their shared students and to integrate the curriculum across courses through, 
for example, joint assignments and readings. Some definitions of learning communities also 
include a pedagogical component, usually focusing on “active engagement,” “active pedagogy,” 
and/or “collaborative learning.” Notably, it is not always clear how the preferred pedagogical 
techniques of learning communities are different from what many consider good pedagogy more 
broadly. Finally, some learning communities include add-on student supports, like enhanced 
counseling, tutoring, or other student services (Minkler, 2002; Richburg-Hayes, Visher, & 
Bloom, 2008; Smith, 2001; Smith et ah, 2004). 

By co-enrolling students into two or more courses, learning communities are intended to 
foster stronger connections among students (Smith et ah, 2004). Through sustained academic 
relationships among students and faculty, students are expected to feel more integrated into a 
community of peers and college life, leading to a greater level of commitment. In addition, 
integrating course materials may help students understand connections between disciplines and 
between what they are learning in school and their personal lives, and in doing so, engage 
students more deeply in learning. Curricular integration may be particularly effective when a 
developmental-level course and college-level course are paired in a learning community since 
this allows students to use the basic skills being taught in developmental classes in their college- 
level coursework. Broadly speaking, learning communities are theorized to lead to improved 
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academic outcomes by fostering stronger connections among students and between students and 
faculty, integrating students into campus life, and providing a more engaging academic 
environment. This is hypothesized to improve students’ academic attainment and increase their 
likelihood of persisting in school. 

Research Design: 

This study uses a random assignment research design. Random assignment creates two 
groups of students that are similar in characteristics that can be measured, such as age and 
gender, and in those that are more difficult to measure, such as tenacity and motivation. As a 
result, subsequent differences in outcomes (for example, credit accumulation) can be attributed 
with a high level of confidence to the program. The random assignment procedure and process 
was designed and controlled by the researchers. 

Data Collection and Analysis: 

This study tracked students for seven years, the longest follow-up period of a learning 
communities evaluation. The study gathered data from a number of sources to track students’ 
academic outcomes. The primary data sources for academic outcomes were transcript data from 
CUNY and degree data provided by the National Student Clearinghouse and are based on data 
from all colleges that report to the Clearinghouse - more than 96% of American colleges and 
universities. To track employment and earnings, we analyzed individual-level employment 
records and aggregate earnings data from the New York State Department of Labor 
Unemployment Insurance Data. 

Findings / Results: 

• Increase in enrollment. Program group students enrolled in higher rates than control 
group students in 8 of the 28 terms (including winter and summer sessions) tracked 
during the study. For example, during the first-year winter session, 60.0 percent of 
program group members enrolled compared to 54.5 percent of control group members, an 
estimated impact of 5.5 percentage points (p=.03 ). See Figure 1. 

• Increase in credits earned. Program group students earned an estimated 4.0 additional 
credits over seven years, a 7.5 percent increase (p=.09). See Table 1. 

• Possible increase in graduation rates. Although not statistically significant, program 
group students’ graduation rate was 3.3 percentage points higher than that of control 
group students: 39.5 percent graduated, compared to only 36.2 percent of control group 
students. This positive estimate, which follows positive estimates in each of the preceding 
four years, is encouraging for an intervention of such a short duration, but the estimated 
effect on graduation is only on the borderline of what is sometimes conventionally 
viewed as statistically significant in year six (p=.l ). See Table 1. 

• Underpowered to detect meaningful effects on earnings. Although this study was not 
designed to be powered to detect effects on employment and earnings, as one of the 
largest scale experiments in higher education history, it provides a unique opportunity to 
reflect on the possibility of connecting higher education interventions to effects on 
employment and earnings. Although not statistically significant, the program’s estimated 
effect on average earnings is over $1,000 during years six and seven. The estimated effect 
represents an increase of around 8 percent over the control group’s base of $ 12,700 in 
year six and $14,700 in year seven. Despite its large sample size, this experiment is not 
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sufficiently powered to detect an effect of $ 1 ,000. The authors find that an adequately 
powered study of an educational program’s impact on employment and earnings would 
require an extraordinarily large sample size or effect size to find impacts (under certain 
assumptions - see conclusion section below). 

Conclusions: 

Our conclusions fall into two categories: first, we offer some lessons learned regarding 
designing a higher education evaluation of an intervention intended to improve future earnings. 
Second, we offer some thought regarding learning communities. 

The authors find that this study, despite its large sample size among higher education 
evaluations, is underpowered to estimate impacts on employment and earnings. To offer insight 
into designing evaluations of college programs aimed at improving economic outcomes, Table 2 
presents scenarios that one might encounter in an evaluation and calculates the minimum 
detectable effect (MDE) required to find a statistically significant impact on earnings. 

The first row of the first panel in Table 2 parallels our study, where n is around 1,500, the 
MDE is around $2,874, and the estimated degree effect is 3.3 percentage points. If effects on 
earnings derive solely from effects on earning a degree, then the first row shows that the 
required returns to a degree would need to be $87,878 in the seventh year, which constitutes a 
592 percent increase over the average earnings in the control group. 

The remainder of Table 2 depicts scenarios for a program whose effects on degree receipt 
are larger than those estimated in this study (5, 10, 15, and 20 percentage points) along with a 
scenario where a study includes 10,000 sample members. With a sample size of 1,500, even a 20 
percentage point effect on degree receipt would require the returns to a degree to be $14,370, or 
98 percent, for the study to be adequately powered. In fact, even in an evaluation with 10,000 
students where an intervention increases degree receipt by 20 percentage points, the return to a 
degree would have to be $5,565, or 38 percent, in order for the study to be adequately powered - 
this remains well above current estimates of the average returns to an associate’s degree. 

The information provided in Table 2 suggests that if a college programs’ effects on 
earnings arise through effects on degrees alone, then identifying the causal effect of an 
educational intervention on earnings is probably more difficult than generally acknowledged. 

As noted above, the promising short-term outcomes for this program led to a five-college 
replication of learning communities for students in developmental education classes. The 
colleges hoped that KCC’s positive outcomes would be replicated at their own schools. 

However, the evaluations of these five learning communities programs suggest that the findings 
discussed here may be a bit of an outlier. While the aggregate effects for the five schools were 
positive on measures such as enrollment and credits earned, the results were quite modest, with 
multiple schools showing no impacts whatsoever. The importance of replication cannot be 
understated: learning communities may produce positive student outcomes in one location but no 
impacts in another, and are not the panacea for low community college graduation rates some 
might hope them to be. 
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LEARNING COMMUNITIES 


Figure 1 . CUNY Enrollment in Main Sessions and Intersessions 



Sources: Calculations from CUNY Institutional Research Database (IRDB). 

Notes: Estimates are adjusted by research cohort. Cluster-robust standard errors are used when calculating p-values - students are clustered by learning 
community link. 

A two-tailed test was applied to impact estimates. Statistical significance levels are indicated as: *** p < .01; ** p < .05; * p < .10. 
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Table 1 . Academic Outcomes, Years One Through Seven 


Outcome 

Program 

Group 

Control 

Group 

Estimated 

Effect 

p-value 

Cumulative credits attempted (at CUNY) 

Year 1 

28.3 

26.9 

1.4 

0.024 

Year 2 

47.4 

44.6 

2.7 

0.024 

Year 3 

58.8 

55.8 

3.0 

0.071 

Year 4 

67.1 

63.1 

4.1 

0.045 

Year 5 

73.4 

68.7 

4.7 

0.041 

Year 6 

77.9 

73.0 

4.8 

0.054 

Year 7 

80.9 

76.4 

4.6 

0.087 

Cumulative credits earned (at CUNY) 

Year 1 

20.3 

18.2 

2.1 

0.002 

Year 2 

33.9 

31.1 

2.8 

0.020 

Year 3 

42.0 

39.4 

2.6 

0.098 

Year 4 

48.1 

44.8 

3.4 

0.076 

Year 5 

52.8 

49.0 

3.7 

0.074 

Year 6 

56.3 

52.3 

4.0 

0.078 

Year 7 

58.8 

54.7 

4.0 

0.092 

Earned a degree (at any college) (%) 

Year 1 

0.0 

0.0 

0.0 


Year 2 

6.1 

5.4 

0.8 

0.615 

Year 3 

20.2 

17.0 

3.2 

0.202 

Year 4 

26.5 

23.8 

2.7 

0.302 

Year 5 

31.9 

28.4 

3.5 

0.210 

Year 6 

35.9 

31.5 

4.4 

0.104 

Year 7 

39.5 

36.2 

3.3 

0.236 

Highest degree earned by Y ear 7 (%) a 

Bachelor's degree or higher 

16.5 

14.8 

1.7 

0.404 

Associate's degree 

22.2 

20.5 

1.7 

0.434 

Sample size (total = 1,534) 

wr~ 

765 




Sources : Calculations from CUNY Institutional Research Database (IRDB) and National Student 
Clearinghouse data. 


Notes'. Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by research cohort. 

Cluster-robust standard errors are used when calculating p-values - students are clustered by 
learning community link. 

A two-tailed test was applied to differences between research groups. 

Cumulative credits include both college-level and developmental credits. Values of zero credits 
attempted/earned have been imputed for 9 students for whom CUNY data were unavailable. 

a Percentage who earned bachelor's degree or higher and percentage who earned associate's degree 
do not add up to total because the degree type of some degree-earners was unknown. 
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Table 2. Necessary Retums-to-Degree, Assuming Earnings Effects Only Through Degree Receipt 


Sample Size 
and MDE 

Hypothetical Degree 
Effect 

(percentage points) 

Necessary 
Returns-to-degree ($) 

Necessary 
Returns-to-degree (%) 

n = 1,500 

3.3 

87,091 

592% 

MDE = $2874 

5.0 

57,480 

391% 


10.0 

28,740 

196% 


15.0 

19,160 

130% 


20.0 

14,370 

98% 

n = 10,000 

3.3 

33,727 

229% 

MDE = $1113 

5.0 

22,260 

151% 


10.0 

11,130 

76% 


15.0 

7,420 

50% 


20.0 

5,565 

38% 


Notes: MDEs assume power is 80 percent, the significance level is 5 percent, and the standard deviation of 
the earnings outcome is $19,880. 





